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Purpose 


Text complexity has received increased attention due to the CCSS!, which call for students to comprehend increasingly more complex texts as they 
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progress through grades. Quantitative text complexity (or readability) indices are based on text attributes (e.g., sentence lengths, and lexical, a 0.35 0.00 0.01 10.07 ; 0150.05 -0.14. 0.31 0.22 0.30 fe Ce 0.09|10.25110.19|10.07 's 
syntactic, & semantic difficulty), quantified by mathematical formulas to estimate text. ; i aeae oF eae ry as 
Researchers have warned that texts should not be created to reach particular readability scores” *; a frequent practice of “equating” ORF passages. Se 0.21 -0.26 0.19 ie 0.28 -0.16 -0.16 0.17. -0.13 || -0.03 “he; sis. 0.60 0.01 
Issues we encountered were: estimates often differ; estimates do not account for student variance; and many require texts of a certain length. S h or t ——— — Z = —— 7 = q : — 7 
The purpose of this study was to provide comparisons of quantitative text complexity estimates based on text features to estimates of “text ee 0.21: i Ay me , 0.12). 0.47 - 0.19 : 
difficulty” based on student performance. Specifically, these comparisons are situated in the context of curriculum-based measurement (CBM) ta! i ieee 1 ee 
assessment of oral reading fluency (ORF), where the passages range from 20 to 105 words. Wee 0.23 aie eid ie 0.54 0.55 0.32 
a mf; ceils ie ji ee) wee ee 
Method ES “all cae 
Method |p = : | ale: | all: 


Participants 


910 students (Grade 2 = 259, Grade 3 = 329, Grade 4 = 322) were assessed online, via laptops in a one-to-one administration, 0.18 |0.11 || -0.01 : 0.28 ; 0.40 ; 0.24 : 
during which each student read approximately 3 long, 5 medium, and 10 short passages. aaa 7 | 
0.28 -0.14 0.01 | “ 0.15 0.12 | -0.12 0.34 0.13 /0.26 0.63} 0.66 0.01 

Passages Passage Words per Number of Mi ed lM kines aaalloda e oS Kini paalode i Kies ae : Kinga aoe : 
We administered 330 ORF passages — 110 at Grades 2, 3,4: Length Passage Passages per Grade sa is t i te 
Each passage was an original work of narrative fiction, and Long ~ 35 0 vs || worM 
targeted readability at the mid-year level for each grade. Medium ~ 50) 30) 0.44 - 4 be 0.97 0.36 

Short ~ 25 60 eat vy ee ie i . : fos i 
Text Complexity ee a le ii me ja eK eal 
Formality*: Many measures inform 5 principal components: narrativity, syntactic simplicity, word concreteness, referential cohesion, deep cohesion. _seey neers , __sepess : | . __seperey oe 
= (referential cohesion + deep cohesion — narrativity - syntactic simplicity - word concreteness) / 5. a. 0.23 -0.14)0.45) 0.27 - 1 0.08 | -0.23) 0.07) -0.04: a 0.12 0.09 0.16 0.47: rn 0.20 10.10 0.29 0.27: 
Automated Readability Index (ARI): based on letter count =a foe 7 lie ) {dP ar _ baal : 
= 4.71 (letters/words) + 0.5(words/sentences) — 21.43 izeneo ae 0.00 0.27 0.42 ae a 0.13 || -0.02 || -0.06 ae | 0.50 0.29 0.30 S 0.80} 0.82 0.03 
Flesch-Kincaid (grade): based on syllable count (less reliable to program than letter count) - | o Lo NO | ae -0.211/-0.34 2 Ee 4 oF -0.03 0.19 |: a PA - -0.14 0.08 : 7 -0.05 : 
= 0.39(average sentence length) + 11.8(average syllables per word) - 15.59 {ome ——— ie Pan | oe ail i CS | _ i t: 
WCPM (words correct per minute): Recorded audio files scored by trained assessors —_ oN _ 0.67 ° fe ae P i a ak 0.63 dre fe oe a, 0.77 
(with the ability to rewind, replay, and adjust audio), — asi 7 = |) ——— = i) tae soa Ir ; ifs alee 
using the same scoring rules as traditional ORF procedures. a . ia hk Ve acted |e yh i is oe oo ; ae Lal 
= (total words read — words read incorrectly) * ( 60 / computer recorded sec duration) — aaa rrr aera ae vapvar ry eee water reed to 


Latent fluency: Bi-factor CFA model included a general factor (fluency) and two group factors (speed and accuracy). Observed scores were at the Formal is b i 
sentence level for each reading: accuracy = percent of words read correctly, speed = average sec per word. Models estimated separately by grade. Py 0.07, 0.08 0.24 0.20 t* 0.04 0.27 0.08 i: 0.26 0.21 0.36 ? 0.06 

. AES) 0.12 |-0.11/ 0.14 0.12| 0.03, -0.11 0.30|-0.17 0.01 agg 0.65 
Analysis ioe N a! 
Spearman's rank correlations were computed between the text complexity estimates. All analyses conducted with R>® ’. All le N gth S ted hed |) ge Kas 0.01.0.04 Ms seat 0.21 0.35 - seca 
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Text Complexity Estimate Not Used Flesch- Latent 3 
Coleman-Liau Index — Based on texts of 100+ words. Grade Length Formalit ARI Kincaid —~WCPM fluency =a 
Gunning FOG Index — Based on texts of 100 + words. y) -0.65 05) 1.87 0.7)12.54107) 56.88 (oD yeee|| uatent 7 Latent 
FORCAST - Based on texts of 150 (or 100) words. 4 ; i ; ; 
Spache — Based on text of 100-150 words & intended to measure only through Grade 3. 3 -0.62 04) 3.27 ©8)]3.58]07) 79.99 (15) a ‘: . “ a ; 
Dale-Chall Readability Formula — Intended to measure Grade 4 and above. 4 0.51 4.90 4.59 93.67 oP at ae 
SMOG -— Intended to measure Grade 4 and above. mae OO: Eh (1.2) | 4. my vee) 
Textkvaluator (ETS) — Ideal for texts of 300+ words. _ | — Short -0.68 0.5) 70.67 41168) 
Eight component scores as sources of comprehension difficulty (academic vocabulary, word unfamiliarity, C | ° 
concreteness, syntactic complexity, lexical cohesion, level of argumentation, degree of narrativity, Med (0.4) 29 55 418.5) TCONCIUSIONS 
interactive/conversational style) W@ FK aligns closest to grade level as an artifact of the passage development M& WCPM and Latent fluency generally increase across lengths and grades 
Lexile — Requirement is based on text of 125+ words. Long (0.3) 86.82 409.2) 0.07 400.2) Oe i Ger naises d t: F ee eer een Grades 3 & 4 BFK and ARI estimat eee ieaoth 
“Early Reading” Lexile — Ideal for Grade 1. estimates increase across grades except: Formality remains fairly Stable across Grades an 7 estimates remain Stable across lengtns 
Nine text characteristics for early-grades text complexity: word structure (decoding demand and number of Short -0.75 ©) 1.85 7 2.54 ©8) 52.12 @&4) -0.13 ©2) W In our study, it was not possible to examine the Latent fluency correlations between grades because they were modeled independently. Also, WCPM and Latent 
Seo re Se Wends) yon Me atie (ane OF ac stot Ayal aetee and WOE arene) ane cena cane 2 Med -0.49 4) 1,90 08) 2.54 06) 61.07 ©) 0.07 (0.1) fluency functioned differently by length; the former with bimodal distributions and the latter with near-normal, which was likely a function of the model choice. 
discourse-level characteristics (intersentential complexity, phrase diversity, text density/ information load, EB Mod lati L f| d WCPM dth | : | d ie ‘thi d 
and noncompressibility). Long -0.58 3) 1,92 06 2.53 06 64.88 (76 0.07 03) rh. erate oe ne ral i. ene an : , and the y sie y a ae _ wit In a es. lex 
e€ measure with the lowest correlations among all measures was Formality = Ferhaps a different dimension of text complexity. 
-(). (0.5) 3, (0.9) 3, (0.8) ; (9.4) -0. (0.3) ‘ : : : ; 
PHO =yer ae aul vepee ve ® Highest correlations observed among ARI, FK, and WCPM by length (which increases by length) = Partly an artifact of passage development, and partly a 
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Mee M010; oun Ant agent a Voie O63) means ftsnactagsocakg OO Short -0.55 6 5.20 42 4.65 a2 86.07 os -0.14 0 — |Discussion 
Dei re ee iernn reene ger veep enn et were rine rhea heen fy pueencrie MU - Does CCSS conflate text complexity with text difficulty!!? If quantitative text complexity is a function of text features, can ORF (wepm, latent fluency) define 
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